How would you design observability — metrics, logs, and traces — for a Go microservice in production?

Expose Prometheus metrics on /metrics, emit structured JSON logs with slog enriched with trace IDs, and instrument with OpenTelemetry for distributed traces. Correlate all three with a shared trace ID.

Observability setup

Observability pillars

1
Metrics: request rate, error rate, latency (P50/P95/P99), goroutine count, GC pause time
2
Logs: structured JSON with traceID, spanID, requestID, userID on every entry
3
Traces: OpenTelemetry spans for HTTP requests, DB queries, outbound calls — visualized in Jaeger/Tempo
4
Correlate by injecting traceID into log context: span.SpanContext().TraceID().String()
5
Stack: Prometheus + Grafana (metrics), Loki (logs), Tempo/Jaeger (traces) — or Datadog/New Relic for managed

Difficulty: 8/10

Topics: metrics collection, structured logging, distributed tracing

Scenario Questions

0-2 years experience

1
Suppose you need to add basic request latency metrics to a Go microservice. Which library would you choose and how would you expose the metrics for Prometheus?
2
If your service is logging JSON lines, what fields would you include to make logs useful for debugging in production?
3
How would you instrument a simple HTTP handler to emit a trace span using OpenTelemetry?

2-5 years experience

1
Your recent deployment caused a spike in error rates, but the logs don’t show any stack traces. Walk me through how you would use metrics, logs, and traces to pinpoint the issue.
2
When adding observability to a new feature, you notice increased CPU usage due to metric collection. How would you balance metric granularity versus overhead?
3
Explain how you would correlate a request’s trace ID across services when the downstream service is written in a different language.

5-8 years experience

1
Design an end‑to‑end observability pipeline for a fleet of Go microservices handling 10k RPS, covering metric aggregation, log storage, and trace sampling. Discuss trade‑offs.
2
Your tracing backend is hitting storage limits during peak traffic. What strategies would you employ to reduce trace volume without losing critical debugging information?
3
How would you ensure that observability instrumentation does not become a source of latency or panic in a high‑throughput Go service?

8+ years experience

1
At a large organization, you need to standardize observability across dozens of teams with heterogeneous tech stacks. How would you define a unified approach for metrics, logs, and traces, and drive adoption?
2
A legacy Go service uses custom logging and no tracing. Outline a migration plan to bring it into the modern observability platform while minimizing risk.
3
Discuss the long‑term maintenance challenges of observability data (retention, schema evolution, cost) and how you’d address them at the architectural level.

Follow-up Questions

❓What would you monitor to detect a slow database query?
❓How do you handle high‑cardinality labels in your metrics?
❓Can you describe how you’d automate alerting based on these signals?